Skip to content

fix(sdks/rust): invoke on_disconnect when mutation is queued after disconnect - #6329

Merged
aasoni merged 6 commits into
clockworklabs:masterfrom
Tyagiquamar:fix/rust-sdk-disconnect-6320
Nov 18, 2026
Merged

aasoni merged 6 commits into
clockworklabs:masterfrom
Tyagiquamar:fix/rust-sdk-disconnect-6320

Conversation

@Tyagiquamar

@Tyagiquamar Tyagiquamar commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes an issue where on_disconnect callbacks were skipped when mutations (such as reducers or procedures) were queued immediately after calling disconnect(), or when the WebSocket sender channel dropped unexpectedly.

Problem

  1. When disconnect() is invoked, PendingMutation::Disconnect is queued into the message pipeline. Any mutation called prior to the mutation queue processing is queued behind Disconnect.
  2. When the run loop applies Disconnect, it cleans up send_chan. The subsequent queued mutation then encounters send_chan == None and returns Err(Error::Disconnected).
  3. The runner loops (run_async, run_threaded, run_background_task) caught Disconnected via error_is_normal_disconnect and returned directly without executing end_connection. As a result, on_disconnect callbacks were never triggered, and callers were left with silent connection termination.
  4. Additionally, if the WebSocket sender loop dropped its receiving channel while send_chan was still populated, calling unbounded_send triggered a panic on .expect(...), poisoning the send_chan mutex and crashing the worker task.

Solution

  • Introduced try_send helper in DbContextImpl that cleanly catches closed channels and routes closure through end_connection without panicking or poisoning the mutex.
  • Updated run_async, run_threaded, and run_background_task to invoke end_connection(None) on normal disconnect errors so on_disconnect is reliably called.

Rollback safety impact

n/a

Issue

Closes #6320

@kim kim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reporting this.

I can see how it is possible for the on_disconnect callback to be skipped. I think this can be fixed with a much smaller diff, however:

fn apply_pending_mutations(&self) -> crate::Result<()> {
    while let Ok(Some(mutation)) =
        get_lock_sync(&self.pending_mutations_recv).try_next()
    {
        match self.apply_mutation(mutation) {
            Err(crate::Error::Disconnected) => {
                return Err(self.end_connection(None));
            }
            result => result?,
        }
    }
    Ok(())
}

Also, instead of expect, just return Error::Disconnected when unbounded_send fails.

Comment thread sdks/rust/src/db_connection.rs Resolved
@Tyagiquamar

Copy link
Copy Markdown
Contributor Author

Updated in 6538803 to adopt the suggested smaller diff:

@bfops
bfops requested a review from kim November 5, 2026 20:58
@kim

kim commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Looks good to me, but I'd like to defer to @aasoni on whether this should merge before #6119 lands, which will also fix the issue.

@kim
kim requested a review from aasoni November 6, 2026 09:44
@Tyagiquamar

Copy link
Copy Markdown
Contributor Author

Thanks @kim ! The head commit (6538803) now implements exactly the shape you suggested: apply_pending_mutations loops on try_next and routes Error::Disconnected through end_connection(None), the inline try_send helper and the run-loop end_connection calls are gone, and each unbounded_send failure returns Error::Disconnected instead of expect. The only red check on the PR is "Check release dependencies", which fails the same way on unrelated PRs, so the branch is otherwise green.

@aasoni aasoni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, I'll rebase the auto-reconnect changes once this lands

@aasoni
aasoni enabled auto-merge November 11, 2026 10:03
@bfops

bfops commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Hi @Tyagiquamar! Can you please redo the commits with verified signatures? Then I think this is good to merge!

@bfops

bfops commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

I'm having our bot redo these commits as verified commits. In the future, we may choose to simply close PRs that do not use verified commits.

@clockwork-labs-bot
clockwork-labs-bot force-pushed the fix/rust-sdk-disconnect-6320 branch from 163baeb to e9e02a3 Compare November 18, 2026 20:31
@aasoni
aasoni added this pull request to the merge queue Nov 18, 2026
Merged via the queue into clockworklabs:master with commit af4b20d Nov 18, 2026
110 of 111 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rust SDK 2.12: on_disconnect never fires if a reducer call is queued right after disconnect()

4 participants